Conventionally, system call wrappers are used frequently in a computing environment to protect security of the computing environment. However, many of these conventional system call wrappers cannot be secure without implementing extensive changes in the underlying operating system. As a result, a significant amount of overhead is incurred.
Furthermore, some conventional system call wrappers are themselves targets for security exploits. The basis for the paradoxical security holes caused by system call wrappers is the violation of assumptions concerning the atomicity of various operations. In the ordinary course of events, user-supplied parameters to system calls are checked by the kernel at the same time that they are going to be used. For example, if a process attempts to open a file for write operations, the kernel may validate the filename against the process capabilities at the same time the kernel reads the file name from the user process address space.
The problem occurs when this implicit contract (simultaneous access and validation) is violated by the appearance of a system call wrapper. Without substantial changes to the operating system, the system call wrapper is forced to leave the user-supplied parameters in the address space of the calling process. So, a process that is capable to change the parameters between the time that the parameters are checked by the system call wrapper and the time that the parameters are used by the kernel can circumvent the protection that was supposed to be supplied by the system call wrapper.
This can be a devastating problem as one typical usage of the system call wrappers is to implement virtual server compartments, where the administrator of any particular compartment is not supposed to have any access (much less administrative access) to anything outside of the compartment. An attacker with legitimate administrative access to one of these compartments (for example, a customer of a web hosting service) may exploit this vulnerability to completely subvert the host system.